Advanced Lane Finding Project¶
The goals / steps of this project are the following:
- Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
1. Compute Camera Calibration using chessboard images¶
Image distortion occurs when a camera looks at 3D objects in the real world and transforms them into a 2D image; this transformation isn’t perfect. Distortion actually changes what the shape and size of these 3D objects appear to be. So, the first step in analyzing camera images, is to undo this distortion so that it we can get correct and useful information out of them
I prepare "object points", which will be the 3D coordinates of the chessboard corners in the world. Here the given chessboard images has fixed on the 2D(9, 6) plane at z=0, such that the object points are the same for each calibration image. Thus, objp is just a replicated array of coordinates, and objpoints will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. imgpoints will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.
I then used the output objpoints and imgpoints to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:
Compute object and image points¶
In [1]:
import numpy as np
import cv2
import glob
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from moviepy.editor import VideoFileClip
#%matplotlib qt
%matplotlib inline
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((6*9,3), np.float32)
objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d points in real world space
imgpoints = [] # 2d points in image plane.
# Make a list of calibration images
images = glob.glob('../camera_cal/calibration*.jpg')
# Step through the list and search for chessboard corners
for fname in images:
img = cv2.imread(fname)
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# Find the chessboard corners
ret, corners = cv2.findChessboardCorners(gray, (9,6),None)
# If found, add object points, image points
if ret == True:
objpoints.append(objp)
imgpoints.append(corners)
# Draw and display the corners
#img = cv2.drawChessboardCorners(img, (9,6), corners, ret)
#cv2.imshow('img',img)
#cv2.waitKey(500)
cv2.destroyAllWindows()
Calibrate the Camera¶
In [2]:
'''
1. Camera Calibration
'''
img = cv2.imread('../camera_cal/calibration1.jpg')
img_size = (img.shape[1], img.shape[0])
# Find the Camera calibration Once given object points, image points(derived
#above line 1-32) and the shape of the grayscale image:
ret, mtx, dist, rvesc, tvecs = cv2.calibrateCamera(objpoints, imgpoints, \
img_size, None, None)
# Above function return distortion coffecient(dist), cameera matrix (mtx)
# and rotation and translation vector in real world and these information will be used
# in section bellow to correct the distortion of an image
undistort = cv2.undistort(img, mtx, dist, None, mtx)
# Visualize the undistortion on a calibration image
f, (ax1, ax2) = plt.subplots(1,2, figsize=(24,9))
f.tight_layout()
ax1.imshow(img)
ax1.set_title('Original Image', fontsize = 30)
ax2.imshow(undistort)
ax2.set_title('Undistorted Image', fontsize = 30)
Out[2]:
2. Provide an example of a distortion-corrected image¶
Use the distortion coffecient, cameera matrix mtx ( to transform 3D objects points to 2D image points), rotation and translation vector calculated above to undistort the test images from the 'test_images' folder by calling the cv2.undistort function as bellow.
In [3]:
'''
Example of a distortion-corrected image.
undistort - undistorts an distorted image.
'''
def undistort(image):
undistorted_image = cv2.undistort(image, mtx, dist, None, mtx)
return undistorted_image
#imread the image in RGB format.
test_img = mpimg.imread('../test_images/test1.jpg')
#undistort the image using
img_undistort = undistort(test_img)
f, (ax1, ax2) = plt.subplots(1,2, figsize=(24,9))
f.tight_layout()
ax1.imshow(test_img)
ax1.set_title('Original Image', fontsize = 30)
ax2.imshow(img_undistort)
ax2.set_title('Undistorted Image', fontsize = 30)
Out[3]:
3. perspective transform with an example of a transformed image.¶
A perspective transform maps the points in a given image to different, desired, image points with a new perspective. This will be useful for calculating the lane curvature later on.
In [4]:
'''
warper - Transform the perspective of a given image using the source and destination points.
'''
def warper(img, src, dst):
# Compute and apply perpective transform
img_size = (img.shape[1], img.shape[0])
#Transform matrix
M = cv2.getPerspectiveTransform(src, dst)
#Inverse transform matrix
Minv = cv2.getPerspectiveTransform(dst, src)
# keep same size as input image
warped = cv2.warpPerspective(img, M, img_size, flags=cv2.INTER_NEAREST)
return warped, M, Minv
The code for my perspective transform includes a function called warper(), which appears in lines 1 through 18 above. The warper() function takes as inputs an image (img), as well as source (src) and destination (dst) points. I chose the hardcode the source and destination points in the following manner:
In [5]:
src = np.float32(
[[(img_size[0] / 2) - 80, img_size[1] / 2 + 120], # top left
[((img_size[0] / 4) - 75), img_size[1]], # bottom left
[(img_size[0] ) - 115, img_size[1]], # bottom right
[(img_size[0] / 2 + 120), img_size[1] / 2 + 120]] # top right
)
dst = np.float32(
[[(img_size[0] / 4), 0],
[(img_size[0] / 4), img_size[1]],
[(img_size[0] * 3 / 4), img_size[1]],
[(img_size[0] * 3 / 4), 0]])
warp_img, M, Minv = warper(img_undistort, src, dst)
# Draw src points on undistorted image
pts = np.array(src, np.int32)
pts = pts.reshape((-1,1,2))
cv2.polylines(test_img,[pts],True,(255,0,0), 3)
# Draw dst points on warp image
pts = np.array(dst, np.int32)
pts = pts.reshape((-1,1,2))
cv2.polylines(warp_img,[pts],True,(255,0,0), 3)
f, (ax1, ax2) = plt.subplots(1,2, figsize=(24,9))
f.tight_layout()
ax1.imshow(test_img)
ax1.set_title('Undistorted Image', fontsize = 30)
ax2.imshow(warp_img)
ax2.set_title('Warped Image', fontsize = 30)
Out[5]:
I verified that my perspective transform was working as expected by drawing the src and dst points onto a test image and its warped counterpart to verify that the lines appear parallel in the warped image as above
4. Color transforms, Gradients or other methods to create a thresholded binary image¶
I used a combination of color and gradient thresholds to generate a binary image (bellow). Althouhg I tried with differen combination such as direction gradient, magnituted gradient, sobel x gradient, HLS color gradient I find sobel x gradient, S-color gradient, L-color gradient from HLS are usefull for this exercise specially the L-color threshold was really usefull in detecting the shadow on the road (from flyover, lane crack repair etc.) and overcome them.
Here's an example of my output for this step.
In [6]:
# Define a function that applies Sobel x and y,
# then computes the direction of the gradient
# and applies a threshold.
def dir_threshold(gray, sobel_kernel=3, thresh=(0, np.pi/2)):
# Assumed given image is already in Gray
# 1) Convert to grayscale
#gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# 2) Take the gradient in x and y separately
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize = sobel_kernel)
sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize = sobel_kernel)
# 3) Take the absolute value of the x and y gradients
sobelx_abs = np.absolute(sobelx)
sobely_abs = np.absolute(sobely)
# 4) Use np.arctan2(abs_sobely, abs_sobelx) to calculate the direction of the gradient
gradient_dir = np.arctan2(sobely_abs, sobelx_abs)
# 5) Create a binary mask where direction thresholds are met
binary_output = np.zeros_like(gradient_dir)
# 6) Return this mask as your binary_output image
binary_output[(gradient_dir > thresh[0]) & (gradient_dir < thresh[1])] = 1
return binary_output
# Define a function to return the magnitude of the gradient
# for a given sobel kernel size and threshold values
def mag_thresh(gray, sobel_kernel=3, mag_thresh=(0, 255)):
# Take both Sobel x and y gradients
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
# Calculate the gradient magnitude
gradmag = np.sqrt(sobelx**2 + sobely**2)
# Rescale to 8 bit
scale_factor = np.max(gradmag)/255
gradmag = (gradmag/scale_factor).astype(np.uint8)
# Create a binary image of ones where threshold is met, zeros otherwise
binary_output = np.zeros_like(gradmag)
binary_output[(gradmag >= mag_thresh[0]) & (gradmag <= mag_thresh[1])] = 1
# Return the binary image
return binary_output
def get_thresholded_binary_image(img, s_thresh=(80, 255), sx_thresh=(10, 80)):
#def get_thresholded_binary_image(img, s_thresh=(100, 255), sx_thresh=(10, 70)):
img = np.copy(img)
# Convert to HLS color space and separate the S channel
hsl = cv2.cvtColor(img, cv2.COLOR_RGB2HLS).astype(np.float)
s_channel = hsl[:,:,2]
l_channel = hsl[:,:,1]
#Calculate sobel gradient along x axis
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY).astype(np.float)
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0)
abs_sobelx = np.absolute(sobelx)
scaled_sobel = np.uint8(255*abs_sobelx/np.max(abs_sobelx))
# Threshold x gradient
sxbinary = np.zeros_like(scaled_sobel)
sxbinary[(scaled_sobel >= sx_thresh[0]) & (scaled_sobel <= sx_thresh[1])] = 1
# Threshold S - color channel
s_binary = np.zeros_like(s_channel)
s_binary[(s_channel >= s_thresh[0]) & (s_channel <= s_thresh[1])] = 1
# Threshold L - color channel
l_binary = np.zeros_like(l_channel)
l_binary[(l_channel >= 120) & (l_channel <= 255)] = 1
#dir_of_gradient
#dir_of_grad = dir_threshold(gray, sobel_kernel=3, thresh=(np.pi/7, np.pi/2))
#magnitude of the gradient
#mag_binary = mag_thresh(gray, sobel_kernel=3, mag_thresh=(30, 100))
# Combine the two binary thresholds
combined_binary = np.zeros_like(sxbinary)
combined_binary[((s_binary == 1) | (sxbinary == 1)) & (l_binary == 1) ] = 1
return combined_binary
thresholded_binary_image = get_thresholded_binary_image(warp_img)
# Plot the result
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(warp_img)
ax1.set_title('Original Image', fontsize=30)
ax2.imshow(thresholded_binary_image, cmap='gray')
ax2.set_title('Thresholded Binary Image', fontsize=30)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
5. Identify lane-line pixels and fit their positions with a polynomial¶
Here I take the histogram of a binary warped image along the y-axis and take half of the histogram to detect left lane pixels and the remaining half to detect right lane pixels. Then use a sliding window to find the non-zero left and right indices of x and y of the detected lane pixels and from there fit the coordinates of x and y into a second order polynomial equation and return the left and right coefficients along with the detected non-zero indices.
In [7]:
#print(thresholded_binary_image.shape)
def detect_lane_lines (binary_warped, draw):
# Take a histogram of the bottom half of the image
# // divide with integer discard remainder
histogram = np.sum(thresholded_binary_image[thresholded_binary_image.shape[0]//2:,:], axis=0)
# this is for debugging, draw = 1 if we need to see how the lane detection goes
if draw == 1:
plt.plot(histogram)
#Create an white output image to draw on and visualize the result
#out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
# Find the peak of the left and right halves of the histogram
# These will be the starting point for the left and right lines
midpoint = np.int(histogram.shape[0]/2)
leftx_base = np.argmax(histogram[:midpoint])
rightx_base = np.argmax(histogram[midpoint:]) + midpoint
# Choose the number of sliding windows
nwindows = 9
# Set height of windows
window_height = np.int(binary_warped.shape[0]/nwindows)
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Current positions to be updated for each window
leftx_current = leftx_base
rightx_current = rightx_base
# Set the width of the windows +/- margin
margin = 50
# Set minimum number of pixels found to recenter window
minpix = 50
# Create empty lists to receive left and right lane pixel indices
left_lane_inds = []
right_lane_inds = []
# Step through the windows one by one
for window in range(nwindows):
# Identify window boundaries in x and y (and right and left)
win_y_low = binary_warped.shape[0] - (window+1)*window_height
win_y_high = binary_warped.shape[0] - window*window_height
win_xleft_low = leftx_current - margin
win_xleft_high = leftx_current + margin
win_xright_low = rightx_current - margin
win_xright_high = rightx_current + margin
# Draw the windows on the visualization image
#cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),(0,255,0), 2)
#cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),(0,255,0), 2)
# Identify the nonzero pixels in x and y within the window
good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & \
(nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]
good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & \
(nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]
# Append these indices to the lists
left_lane_inds.append(good_left_inds)
right_lane_inds.append(good_right_inds)
# If you found > minpix pixels, recenter next window on their mean position
if len(good_left_inds) > minpix:
leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
if len(good_right_inds) > minpix:
rightx_current = np.int(np.mean(nonzerox[good_right_inds]))
# Concatenate the arrays of indices
left_lane_inds = np.concatenate(left_lane_inds)
right_lane_inds = np.concatenate(right_lane_inds)
# Extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit a second order polynomial to each
left_fit = np.array([0,0,0], dtype='float')
right_fit = np.array([0,0,0], dtype='float')
if len(leftx) != 0:
left_fit = np.polyfit(lefty, leftx, 2)
if len(rightx) != 0:
right_fit = np.polyfit(righty, rightx, 2)
return left_lane_inds, right_lane_inds, left_fit, right_fit
In [8]:
# now call the function to try the detection
left_lane_inds, right_lane_inds, left_fit, right_fit = detect_lane_lines(thresholded_binary_image, 1)
In [9]:
'''
When the lanes are previously detected, this function can be used to find the left and right coefficients of the
fitted second order equation.
'''
def use_previous_detection (binary_img, l_fit, r_fit):
nonzero = binary_img.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Set the width of the windows +/- margin
margin = 50
# Create empty lists to receive left and right lane pixel indices
#left_lane_inds = []
#right_lane_inds = []
# find left indices based on previous left fit
left_lane_inds = ((nonzerox > (l_fit[0]*(nonzeroy**2) + l_fit[1]*nonzeroy + l_fit[2] - margin)) &
(nonzerox < (l_fit[0]*(nonzeroy**2) + l_fit[1]*nonzeroy + l_fit[2] + margin)))
# find right indices based on previous right fit
right_lane_inds = ((nonzerox > (r_fit[0]*(nonzeroy**2) + r_fit[1]*nonzeroy + r_fit[2] - margin)) &
(nonzerox < (r_fit[0]*(nonzeroy**2) + r_fit[1]*nonzeroy + r_fit[2] + margin)))
# Now derive left and right line pixel positions based on the above indices
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
# Fit the second order polynomial for left and right lane based on above pixels
if len(leftx) != 0:
# Fit a second order polynomial to each
l_fit = np.polyfit(lefty, leftx, 2)
if len(rightx) != 0:
r_fit = np.polyfit(righty, rightx, 2)
return left_lane_inds, right_lane_inds, l_fit, r_fit
Draw the line through the detected lane pixels using the above fitted coefficients(A, B and C) for left and right fit.
In [10]:
def draw_lanes_and_lines(binary_warped, left_lane_inds, right_lane_inds, left_fit, right_fit):
# Create an image to draw on lane pixels and detected lines
out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# draw the left lane pixels with red
out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
# draw the right lane pixels with blue
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
# Generate x and y values for plotting
ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0])
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
#draw the image
plt.imshow(out_img)
#draw the fitted lane lines
plt.plot(left_fitx, ploty, color='green')
plt.plot(right_fitx, ploty, color='green')
draw_lanes_and_lines(thresholded_binary_image, left_lane_inds, right_lane_inds, left_fit, right_fit)
6. Calculated the radius of curvature of the lane and the position of the vehicle with respect to center.¶
I use the following equation to find the radius of curvature for detected left and right lanes f'(y)= dx/dy = 2Ay+B f′′(y)= d2x/dy2 = 2A and Radius of curve = ((1+(2Ay+B)2)1.5)/|2A|
Given the binary warped image, left and right indices of detected lanes and the left and right coefficients of the fitted lines, the corresponding x and y coordinates in pixel value were derived. Then use the conversion ratio (from pixel to meter) devive the fitted coefficients(A, B, C) again in meters and from there use the equation above to derive radius of curvature for left and right lane in meter. Aslo from bottom Ys (left and right of the image,close to car dashboard) find the Xs coordinate and derive the middle lane line. Then find the position of the vehicle by taking the absolute difference between the middle of image and the middle lane line above. I also return the bottom left and right x-coordinate of the lanes to find the space between the lanes which will be used to correct the detection later on.
In [11]:
'''
Calculated the radius of curvature of the lane and the position of the
vehicle with respect to center.
'''
def find_radius_of_curvature(binary_warped, left_lane_inds, right_lane_inds, \
left_fit, right_fit):
y_bottom = binary_warped.shape[0]
# to cover same y-range as image
ploty = np.linspace(0, y_bottom - 1, y_bottom)
# maximum y-value, corresponding to the bottom of the image
y_eval = np.max(ploty)
# Define conversions in x and y from pixels space to meters
ym_per_pix = 30/720 # meters per pixel in y dimension
xm_per_pix = 3.7/700 # meters per pixel in x dimension
# Identify the x and y positions of all nonzero pixels in the image
nonzero = binary_warped.nonzero()
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Extract left and right line pixel positions
leftx = nonzerox[left_lane_inds]
lefty = nonzeroy[left_lane_inds]
rightx = nonzerox[right_lane_inds]
righty = nonzeroy[right_lane_inds]
#leftx = leftx[::-1] # Reverse to match top-to-bottom in y
#rightx = rightx[::-1] # Reverse to match top-to-bottom in y
# Fit a second order polynomial to each, this is to draw
# left and right lane through the detected lane pixels
left_fitx = left_fit[0]*lefty**2 + left_fit[1]*lefty + left_fit[2]
right_fitx = right_fit[0]*righty**2 + right_fit[1]*righty + right_fit[2]
'''
# Plot the detected lanes (left and right)
mark_size = 1
plt.plot(leftx, lefty, 'o', color='red', markersize=mark_size)
plt.plot(rightx, righty, 'o', color='blue', markersize=mark_size)
#plt.xlim(0, 1280)
#plt.ylim(0, 720)
# Plot the fitted left and right line
plt.plot(left_fitx, lefty, color='green', linewidth=1)
plt.plot(right_fitx, righty, color='green', linewidth=1)
'''
left_curverad = 0
right_curverad = 0
left_fit_cr = np.array([0,0,0], dtype='float')
right_fit_cr = np.array([0,0,0], dtype='float')
# fit again with real dimention
if len(leftx) != 0:
left_fit_cr = np.polyfit(lefty*ym_per_pix, leftx*xm_per_pix, 2)
if len(rightx) != 0:
right_fit_cr = np.polyfit(righty*ym_per_pix, rightx*xm_per_pix, 2)
# Calculate the new radii of curvature
if len(left_fit_cr) != 0:
left_curverad = ((1 + (2*left_fit_cr[0]*y_eval*ym_per_pix + \
left_fit_cr[1])**2)**1.5) / np.absolute(2*left_fit_cr[0])
if len(right_fit_cr) != 0:
right_curverad = ((1 + (2*right_fit_cr[0]*y_eval*ym_per_pix + \
right_fit_cr[1])**2)**1.5) / np.absolute(2*right_fit_cr[0])
# Camera position (middle of the car) is middle of the image
# along the x - axis
camera_pos = binary_warped.shape[1]/2
left_bottom_x = left_fit[0]*y_bottom**2 + left_fit[1]*y_bottom + left_fit[2]
right_bottom_x = right_fit[0]*y_bottom**2 + right_fit[1]*y_bottom + right_fit[2]
lane_center_pos = (left_bottom_x + right_bottom_x)/2
camera_dis_from_center = (camera_pos - lane_center_pos)*xm_per_pix
# radius of curvature, center distance is in meters
return left_curverad, right_curverad, camera_dis_from_center, left_bottom_x, right_bottom_x
l_curved, r_curved, center_distance, l_bottom_x, r_bottom_x = find_radius_of_curvature(thresholded_binary_image, \
left_lane_inds, right_lane_inds, left_fit, right_fit)
print(l_curved, 'm', r_curved, 'm', center_distance, 'center distance', 'bottom left x:', l_bottom_x, \
'bottom right x:', r_bottom_x)
7. Provide an example image of the result plotted back down onto the road such that the lane area is identified clearly.¶
In [12]:
def draw_lines(original_img, warped, left_fit, right_fit, Minv):
copy_orig_img = np.copy(original_img)
y,x = warped.shape
# Cover same y-range as image
ploty = np.linspace(0, y-1, num=y)
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
# Create an image to draw the lines on
warp_zero = np.zeros_like(warped).astype(np.uint8)
color_warp = np.dstack((warp_zero, warp_zero, warp_zero))
# Recast the x and y points into usable format for cv2.fillPoly()
pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
pts = np.hstack((pts_left, pts_right))
# Draw the lane onto the warped blank image
cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))
# Warp the blank back to original image space using inverse perspective
# matrix (Minv)
newwarp = cv2.warpPerspective(color_warp, Minv, (x, y))
# Combine the result with the original image
result = cv2.addWeighted(copy_orig_img, 1, newwarp, 0.3, 0)
#plt.imshow(result)
return result
In [13]:
output = draw_lines(test_img, thresholded_binary_image, left_fit, right_fit, Minv)
output = cv2.cvtColor(output, cv2.COLOR_BGR2RGB)
plt.imshow(output)
Out[13]:
Draw some usefull information such as left and right lane curvature, space between lanes etc. This was visually observed to tune the detection and draw the lines.
In [14]:
def draw_text(img, l_curv, r_curv, center_distance, bottom_x_diff):
font = cv2.FONT_HERSHEY_SIMPLEX
text1 = 'Radius of Curvature: ' + 'left {:04.2f}'.format(l_curv) + 'm' \
+ ' Right {:04.2f}'.format(r_curv) + 'm'
cv2.putText(img, text1,(40,70), font, 0.5 ,(255,255,255),2,cv2.LINE_AA)
text2 = 'Center distance: ' + '{:04.2f}'.format(center_distance) + 'm ' + \
'Lane separation: ' + '{:04.2f}'.format(bottom_x_diff)
cv2.putText(img, text2,(40,120), font, 0.5 ,(255,255,255),2,cv2.LINE_AA)
return img
8. Pipeline to process the image.¶
Here for each frame (input) of the video first it undistorts the image, do a perspective transform , then apply color and gradient thresholds to create a binary image. This function also return the inverse perspective transform matrix which will be used to draw the lanes.
In [15]:
def pipeline(img):
# Undistort
undist_img = undistort(img)
# Do a perspective Transform
warp_img, M, Minv = warper(undist_img, src, dst)
# Threshold HLS S-channel and x Gradient
thresholded_binary_image = get_thresholded_binary_image(warp_img)
return thresholded_binary_image, Minv
Define a class to receive the characteristics of each line detection¶
In [16]:
class Line():
def __init__(self):
# was the line detected in the last iteration?
self.detected = False
# x values of the last n fits of the line
self.recent_xfitted = []
#average x values of the fitted line over the last n iterations
self.bestx = None
#polynomial coefficients averaged over the last n iterations
self.best_fit = np.array([0,0,0], dtype='float')
#polynomial coefficients for the most recent fit
self.current_fit = []
#radius of curvature of the line in some units
self.radius_of_curvature = None
#distance in meters of vehicle center from the line
self.line_base_pos = None
#difference in fit coefficients between last and new fits
self.diffs = np.array([0,0,0], dtype='float')
#x values for detected line pixels
self.allx = None
#y values for detected line pixels
self.ally = None
'''
Append the passed fit coefficient into last and calculate the
average fit_coefficient, store it in self.best_fit
'''
def add_fit(self, fit_coeff, lane_inds):
self.current_fit.append(fit_coeff)
self.best_fit = np.average(self.current_fit, axis=0)
'''
Remove the last fit coefficient from current fit and
average the remaining and store it in self.best_fit
'''
def rm_fit(self):
no_of_fit = len(self.current_fit)
if no_of_fit > 1:
# remove the last fit.
self.current_fit = self.current_fit[:no_of_fit - 1]
self.best_fit = np.average(self.current_fit, axis=0)
self.detected = False
'''
If there are more than 7 fit coefficients on current_fit
remove the old one (the first one ), fifo/
Average the remaining and store it in self.best_fit
'''
def update_fit(self, fit_coeff, lane_inds):
if len(self.current_fit) > 7:
self.current_fit = self.current_fit[-5:]
self.detected = True
self.best_fit = np.average(self.current_fit, axis=0)
In [17]:
def process_image(img):
copy_img = np.copy(img)
binary_img, Minv = pipeline(copy_img)
if left_obj.detected is False or right_obj.detected is False:
#print("Detecting lanes")
left_lane_inds, right_lane_inds, left_fit, right_fit = \
detect_lane_lines(binary_img, 0)
else:
print("Use previous detected lanes")
left_lane_inds, right_lane_inds, left_fit, right_fit = \
use_previous_detection(binary_img, left_obj.best_fit, right_obj.best_fit)
left_obj.add_fit(left_fit, left_lane_inds)
right_obj.add_fit(right_fit, right_lane_inds)
l_curved, r_curved, center_distance, l_bottom_x, r_bottom_x = find_radius_of_curvature( \
binary_img, left_lane_inds, \
right_lane_inds, left_obj.best_fit, \
right_obj.best_fit)
bottom_x_diff = abs(r_bottom_x - l_bottom_x)
#
# Following need to be true for a valid detection,
# 1. The space between both lanes need to be within 400 to 500 pixel.
# 2. ratio of both curved need to be less than 1.5 (approximate ), roughly parallel
# curve ratio 1.3
if 400 < bottom_x_diff <= 630:
if (((l_curved > r_curved) and (l_curved/r_curved < 1.5)) or \
((r_curved > l_curved) and (r_curved/l_curved < 1.5))):
# Good detection and keep using this fit coefficient for next frame processing
left_obj.update_fit(left_fit, left_lane_inds)
right_obj.update_fit(right_fit, right_lane_inds)
else:
#print("bad fit 1")
# remove this fit coefficient and update the average
left_obj.rm_fit()
right_obj.rm_fit()
else:
#print("bad fit 2")
# remove this fit coefficient and update the average
left_obj.rm_fit()
right_obj.rm_fit()
if left_obj.best_fit is not None and right_obj.best_fit is not None:
print("Drawing...")
output = draw_lines(copy_img, binary_img, left_obj.best_fit, \
right_obj.best_fit, Minv)
l_curved, r_curved, center_distance, l_bottom_x, r_bottom_x = find_radius_of_curvature( \
binary_img, left_lane_inds, \
right_lane_inds, left_obj.best_fit, \
right_obj.best_fit)
bottom_x_diff = abs(r_bottom_x - l_bottom_x)
output = draw_text(output, l_curved, r_curved, center_distance, bottom_x_diff)
else:
output = copy_img
return output
In [18]:
left_obj = Line()
right_obj = Line()
video_output1 = '../output_images/project_video_output.mp4'
video_input1 = VideoFileClip('../project_video.mp4') #.subclip(0, 2)
processed_video = video_input1.fl_image(process_image)
%time processed_video.write_videofile(video_output1, audio=False)
In [19]:
left_obj = Line()
right_obj = Line()
video_output2 = '../output_images/challenge_video_output.mp4'
video_input2 = VideoFileClip('../challenge_video.mp4')#.subclip(0, 5)
processed_video2 = video_input2.fl_image(process_image)
%time processed_video2.write_videofile(video_output2, audio=False)
In [ ]:
In [ ]: